Columbia Newsblaster: Multilingual News Summarization on the Web

We propose to show the new multilingual version of the Columbia Newsblaster news summarization system. The system addresses the problem of user access to browsing news in multiple languages from multiple sites on the internet. The system automatically collects, organizes, and summarizes news in multiple source languages, allowing the user to browse news topics with English summaries, and compare perspectives from different countries on the topics. 1 Multilingual Columbia Newsblaster We propose to present a demo of the multilingual version of Columbia Newsblaster which is currently in development. Columbia Newsblaster1 (McKeown et al., 2002) is a system for news browsing that crawls news from the web, clusters related articles, summarizes the multidocument clusters, and presents the clusters in a browsing interface separated into pre-defined categories. In this demo, participants will be able to • Browse summaries of current news from multiple languages • View news clusters with documents from a particular language • Compare summaries from documents in different languages The Multilingual version of Columbia Newsblaster is built upon the English version of Columbia Newsblaster, taking advantage of the existing system by translating documents into English. The system has six major phases: crawling, article extraction, clustering, summarization, classification, and interface generation. In http://newsblaster.cs.columbia.edu/ Figure 1: Multilingual CU Newsblaster Architecture. the demo, we will show the new translation phase, and changes made to encode all text in UTF8 Unicode. Figure 1 depicts the multilingual Columbia Newsblaster architecture, highlighting the changes made to support multiple languages. The multilingual version of Columbia Newsblaster crawls web sites in foreign languages and extracts article text from the HTML pages. Non-English documents are translated and clustered with English documents using the existing document clustering system. For translation we use an interface to the babelfish translation system, and an interface to a statistical Arabic machine translation system from IBM for Arabic documents. The resulting document clusters are then summarized, with different summaries created for documents based on country and language. Extracting article text Our previous approach to extracting article text was hand crafted for English sites; to support non-English sites we incorporated a new article extraction module that uses machine learning techniques to identify the article text. The new module parses HTML into blocks of text and computes a set of 34 simple text features for each text block. Training data is generated using a GUI, and Ripper (Cohen, 1996) is used to induce a hypothesis for categorizing text. The is currently working with sites in English, Russian, Japanese, Chinese, French, Spanish, GerFigure 2: A screen shot comparing a summary from English documents to a summary from German documents. man, Italian, Portuguese, Korean, and Arabic. Over an English training set composed of 353 articles, the extractor had 89% recall and 90% precision. It had similar performance for Russian and Japanese training sets. Using rules learned for one language on a different data set significantly degraded performance, showing that the system is able to adapt to different sites and languages. Adding new sites is easy; a human annotates web pages using the GUI, and a new categorization hypothesis is learned from the new training data.